1 Using Convex Pseudo - Data to Increase Prediction Accuracy

A prediction algorithm is consistent if given a large enough sample of instances from the underlying distribution, it can achieve nearly optimal generalization accuracy. In practice, the training set is finite and does not give an adequate representation of the underlying distribution. Our work is based on a simple method for generating additional data from the existing data. Using this new data (convex pseudo-data) it is shown empirically that on a variety of data sets prediction accuracy of an algorithm can be significantly improved. This is shown first in classification using the CART algorithm. Similar results are shown in regression. Then pseudo-data is applied to bagging CART. Although CART is being used as a test bed, the idea of generating convex psuedo-data can be applied to any prediction method. Given a training set T={(y n ,x n), n=1, ... ,N} where the y are either class labels or numerical values and the x are M-dimensional input vectors, consider an algorithm that uses T to construct a predictor h(x) of future y-values given the input values x. If the algorithm is consistent i.e. CART, C4.5, Neural Nets, then the larger the size N of the training set, the smaller the generalization error. One dream, assuming the training set consists of independent draws from the same underlying distribution P(dy,dx) is that we can keep drawing as long as we want-constructing a large training set. But generally, we are confined to a training set of given size. One way around this restriction is to try and manufacture new data from the given training set. Another approach is to use T to estimate P(dy,dx) (Shang and Breiman[1996b]). Using the estimated probability distribution, an infinite number of instances can be generated. This method gave promising results but ran into serious technical difficulties. Kernel density estimates were used for the numerical estimates, and ad hoc methods for categorical values. Numerous parameters had to be estimated to plug into the probability estimate, and the result was an complex and unwieldy procedure. Our method for generating new data from the old is relatively simple and depends only on a single parameter d, 0<d<1. To create a new data instance, these steps are followed: I) Select two instances (y,x) , (y',x') at random from the training set. ii) Select a random number v from the interval [0,d], and let u=1-v. iii) The new instance is (y'',x'') where y''=y, and …